Detecting input based on multiple gestures

ABSTRACT

Detecting user input based on multiple gestures is provided. One or more interactions are received from a user via a user interface. An inferred interaction is determined based, at least in part, on a geometric operation, wherein the geometric operation is based on the one or more interactions. The inferred interaction is presented via the user interface. Whether a confirmation has been received for the inferred interaction is determined.

BACKGROUND OF THE INVENTION

The present invention relates generally to the field of accessibility,and more particularly to detecting input based on multiple gestures.

In computing, an input device is any peripheral (piece of computerhardware equipment) used to provide data and control signals to aninformation processing system, such as a computer or other informationappliance. A keyboard, a mouse, a scanner, a digital camera and ajoystick are each an example of an input device.

A touchscreen is an electronic visual display that the user controlsthrough simple or multi-touch gestures by touching the screen with aspecial stylus and/or one or more fingers. The user uses the touchscreento react to what is displayed and to control how it is displayed, forexample by increasing the text size.

A user interface is the system by which users interact with a machine,such as a computer. The user interface may include hardware and softwarecomponents. A user interface enables a computer program to receive inputfrom a user and provide output to the user.

SUMMARY

According to one embodiment of the present disclosure, a method fordetecting input based on multiple gestures is provided. The method mayinclude receiving, by one or more processors, one or more interactionsfrom a user via a user interface. The method may further includedetermining, by one or more processors, an inferred interaction based,at least in part, on a geometric operation, wherein the geometricoperation is based on the one or more interactions. The method mayfurther include presenting, by one or more processors, the inferredinteraction via the user interface. The method may further includedetermining, by one or more processors, whether the inferred interactionis confirmed.

According to another embodiment of the present disclosure, a computerprogram product comprising a computer readable storage medium havingprogram instructions embodied therewith, the program instructionsexecutable by a processor to cause the processor to perform a method fordetecting input based on multiple gestures is provided. The method mayinclude receiving, by one or more processors, one or more interactionsfrom a user via a user interface. The method may further includedetermining, by one or more processors, an inferred interaction based,at least in part, on a geometric operation, wherein the geometricoperation is based on the one or more interactions. The method mayfurther include presenting, by one or more processors, the inferredinteraction via the user interface. The method may further includedetermining, by one or more processors, whether the inferred interactionis confirmed.

According to another embodiment of the present disclosure, a computersystem comprising a memory and a processor in communication with thememory, wherein the computer system is configured to perform a methodfor detecting input based on multiple gestures is provided. The methodmay include receiving, by one or more processors, one or moreinteractions from a user via a user interface. The method may furtherinclude determining, by one or more processors, an inferred interactionbased, at least in part, on a geometric operation, wherein the geometricoperation is based on the one or more interactions. The method mayfurther include presenting, by one or more processors, the inferredinteraction via the user interface. The method may further includedetermining, by one or more processors, whether the inferred interactionis confirmed.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a functional block diagram illustrating a computingenvironment, in accordance with an embodiment of the present disclosure;

FIG. 2 is a flowchart depicting operations of an input program, on acomputing device within the environment of FIG. 1, in accordance with anembodiment of the present disclosure;

FIG. 3 is an illustrative example of a user interface, on a computingdevice within the environment of FIG. 1, in accordance with anembodiment of the present disclosure; and

FIG. 4 depicts a block diagram of components of the computing deviceexecuting the input program, in accordance with an embodiment of thepresent disclosure.

DETAILED DESCRIPTION

Some embodiments of the present invention recognize that, in manysituations and applications, a user of a computing device interacts withan element of an interface. The user may interact with an element byperforming a gesture such as, for example, pressing, activating an inputdevice (e.g., clicking a mouse), touching, tapping, or swiping. A motordifficulty, dispositional impairment, or environmental condition impedesthe accuracy of an interaction. Such an impedance may be permanent ortemporary with respect to the affected user. Examples of impedancesinclude, without limitation, a bumpy bus ride, a medical conditioncausing reduced motor dexterity, or the small size of an interactiveelement relative to the size of a user's finger. Similarly, a user mayhave a need to interact with a precision greater than can be attainedwith a single gesture. For example, a surgeon operating a high-precisionremote surgery system may need to interact more precisely than ispossible with a single gesture.

Some embodiments of the present invention provide for detecting inputbased on multiple gestures. Some embodiments determine a geometricaverage based on the multiple interaction, each interaction resultingfrom a gesture by a user. Some embodiments determine an inferredinteraction based on statistical information. In an example of oneembodiment, a user makes multiple gestures, each being an attempt at aninteraction. Based on the attempts, an inferred interaction isdetermined. The inferred interaction is presented to the user, whoconfirms or rejects the inferred interaction.

The present disclosure will now be described in detail with reference tothe Figures. FIG. 1 is a functional block diagram illustrating acomputing environment, generally designated 100, in accordance with anembodiment of the present disclosure. Computing environment 100 includescomputing device 102 connected to network 120. Computing device 102includes interface 104 and input program 106.

In various embodiments of the present invention, computing device 102 isa computing device that can be a standalone device, a server, a laptopcomputer, a tablet computer, a netbook computer, a personal computer(PC), or a desktop computer. In another embodiment, computing device 102represents a computing system utilizing clustered computers andcomponents to act as a single pool of seamless resources. In general,computing device 102 can be any computing device or a combination ofdevices with access to and/or capable of executing interface 104 andinput program 106. Computing device 102 may include internal andexternal hardware components, as depicted and described in furtherdetail with respect to FIG. 4.

In this exemplary embodiment, interface 104 and input program 106 arestored on computing device 102. However, in other embodiments, one orboth of interface 104 and input program 106 may be stored externally andaccessed through a communication network, such as network 120. Network120 can be, for example, a local area network (LAN), a wide area network(WAN) such as the Internet, or a combination of the two, and may includewired, wireless, fiber optic or any other connection known in the art.In general, network 120 can be any combination of connections andprotocols that will support communications with computing device 102, inaccordance with a desired embodiment of the present invention.

Interface 104 is a user interface that includes at least one interfaceelement. In one embodiment, interface 104 is presented to a user ofcomputing device 102, who can interact with interface 104. For example,the user interacts with interface 104 to indicate a selection of one ormore interface elements. Each interface element defines a region ofinterface 104. The user can attempt to interact with an interfaceelement, but instead (e.g., due to an impedance, as discussed above)interacts with another region of interface 104. For example, the userattempts to press, but misses, a button of interface 104. In oneembodiment, interface 104 resides on computing device 102. In otherembodiments, interface 104 may reside on another computing device,provided that interface 104 can access and is accessible by inputprogram 106. In various embodiments, interface 104 is a user interfacegenerated or presented by any of various applications, programs, orsoftware. For example, a user application operating on computing device102 generates and presents interface 104 in order to present informationand receive interactions from a user of computing device 102.

In various embodiments, interface 104 is a one-, two-, orthree-dimensional user interface. For example, interface 104 is aone-dimensional interface including a lever that a user positions. Inanother example, interface 104 is a two-dimensional interface such as agraphical user interface generated by a computer application. In yetanother example, interface 104 is a three-dimensional interface, such asa surgical planning system utilizing physical manipulation of tangibleobjects. Alternatively, interface 104 is a three-dimensional interfacesimulated on a two-dimensional display utilizing computer graphics. Inone embodiment, interface 104 is presented on a display (e.g., display420).

Input program 106 operates to detect input based on multiple gestures.Input program 106 receives one or more interactions. Input program 106determines inferred coordinates based on the one or more interactions.Input program 106 determines an inferred interaction. Input program 106determines a confidence score for the inferred interaction. Inputprogram 106 determines whether the confidence score is above athreshold. If input program 106 determines that the confidence score isabove the threshold, then input program 106 presents the inferredinteraction. If input program 106 determines that the confidence scoreis not above the threshold, then input program 106 receives additionalinteractions. Input program 106 determines whether confirmation of theinferred interaction has been received. If input program 106 determinesthat confirmation has been received, then input program 106 performs theconfirmed interaction. If input program 106 determines that confirmationhas not been received, then input program 106 receives additionalinteractions. In one embodiment, input program 106 resides on computingdevice 102. In other embodiments, input program 106 may reside onanother computing device, provided that input program 106 can access andis accessible by interface 104.

FIG. 2 is a flowchart depicting operations of an input program, on acomputing device within the environment of FIG. 1, in accordance with anembodiment of the present disclosure. For example, FIG. 2 depictsoperations 200 of input program 106, on computing device 102 withincomputing environment 100 of FIG. 1.

In operation 202, input program 106 receives one or more interactions.In one embodiment, input program 106 receives one or more interactionsfrom a user of computing device 102 via interface 104. In anotherembodiment, input program 106 receives interactions from a user ofcomputing device 102 by monitoring and intercepting user interactionswith interface 104. Each of the one or more interactions includes a setof coordinates identifying a location of the interaction. The set ofcoordinates include a value for each dimension of the user interface(e.g., interface 104). For example, in a two-dimensional interface, theset of coordinates (x, y) represent the location on the two-dimensionalinterface of the interaction. In one embodiment, each interaction islocated within the boundaries of a region of interface 104 defined byzero or more interface elements.

In some embodiments, an interaction includes more than one set ofcoordinates. For example, input program 104 receives an interactionresulting from a user performing a continuous gesture (e.g., a swipe).In this case, the interaction indicates sets of coordinates forlocations along the path of the continuous gesture.

In some embodiments, input program 106 receives the one or more inputswithin an input window. The input window is a time period ofpre-determined length since input program 106 last received aninteraction. The time period can be of any length. In one example, theinput window is a period of two seconds. In this example, twointeractions received five seconds apart are not deemed to be attemptsat the same intended interaction. Conversely, in this example, anynumber of interactions is deemed to be attempts at the same intendedinteraction if each is received within two seconds of receiving theprevious interaction. In various embodiments, the input window ispre-programmed, user-configurable, dynamic, or a combination thereof. Inone embodiment, the input window is of variable length dependent on asignal from an accelerometer (not shown) of computing device 102. In anexample, the input window is longer when computing device 102 detects anunsteady environment, such as while the user is on a bus, train, orother transportation, or while the user is walking or jogging.

In operation 204, input program 106 determines inferred coordinatesbased on the one or more interactions. The inferred coordinates includeone or more sets of coordinates that each represents a location withininterface 104. In one embodiment, input program 106 determines theinferred coordinates based on a geometrical average of the one or morereceived interactions (see operation 202). In one embodiment, inputprogram 106 determines each dimension of the inferred coordinates byaveraging the corresponding dimensions of each of the one or morereceived interactions. For example, input program 106 determines theinferred coordinates for a tapping gesture of a three-dimensionalinterface by determining a set of coordinates for which the x, y, and zdimensions respectively equal the averages of the x, y, and z dimensionsof the received interactions.

Input program 106 determines the inferred coordinates for a continuousgesture (e.g., a swipe) based on the one or more sets of coordinates ofeach received interaction. For example, interface program 106 determinesinferred coordinates for a continuous gesture that follows a straightline. In this case, the inferred coordinates and the one or morereceived interactions each have a first set of coordinates representinga starting point of the gesture and a second set of coordinatesrepresenting an ending point of the gesture. Input program 106determines each dimension of the first set of coordinates of theinferred coordinates by averaging the corresponding dimensions of thefirst set of coordinates of each of the one or more receivedinteractions. Similarly, input program 106 determines each dimension ofthe second set of coordinates of the inferred coordinates by averagingthe corresponding dimensions of the second set of coordinates of each ofthe one or more received interactions. In another embodiment, inputprogram 106 determines inferred coordinates of a continuous gestureutilizing one or more statistical and mathematical techniques (e.g.,regression analysis, curve fitting, statistical smoothing, lineinterpolation), based on the one or more sets of coordinates of eachreceived interaction.

In some embodiments, one or more of the dimensional values utilized indetermining the geometric average are weighted. For example, the weightof a value increases or decreases by multiplying the value by a numbergreater than one or less than one, respectively. In one embodiment, sucha multiplier is applied prior to performing the geometric average of thedimensional values. In some embodiments, a dimension of an interactionthat is proximate (e.g., within five percent) of the edge of interface104 is weighted more heavily. The proximity of a set of coordinates tothe edge of the interface is determined based, in part, on theresolution of the interface. For example, interface 104 has a resolutionof 800 horizontal pixels by 600 vertical pixels. For an interactionoccurring at coordinates (775, 300), the x dimension is less than 3.2%from the edge of interface 104. Thus, in a geometric average based, atleast in part, on the interaction, the x value of the interaction isweighted. The weight applied increases the closer the dimension is tothe edge of an interface. In one example, a dimension within 10% of theedge of interface 104 is multiplied by 1.2, while a dimension within twopercent of the edge is multiplied by 2.

In operation 206, input program 106 determines an inferred interaction.In one embodiment, input program 106 determines an inferred interactionby identifying the interface element of interface 104 nearest to thelocation of the inferred coordinates. Input program 106 identifies thenearest interface element by identifying an interface element that hasboundaries encompassing the inferred coordinates. In another embodiment,input program 106 identifies the nearest interface element based on thedistance between the inferred coordinates and one or more interfaceelements. In one embodiment, the distance between the inferredcoordinates and an interface element is the geometric difference betweenthe inferred coordinates and the coordinates of a point along a borderof the interface element (e.g., the point along the border closest tothe inferred coordinates). In another embodiment, the distance is thegeometric difference between the inferred coordinates and another pointcorresponding to the interface element, such as the center-point of theinterface element.

In some embodiments, input program 106 determines an inferredinteraction based on statistical information of prior interactions. Inone embodiment, such statistical information includes measures relatedto, for example, a count of interactions for one or more interfaceelements, a frequency of interactions with an interface element relativeto one or more other interface elements, or a probability thatparticular coordinates correspond to an interface element. In variousembodiments, the statistical information relates to one or more users.In one embodiment, the statistical information relates to priorinteractions of the user upon whose interactions input program 106 isdetermining an inferred interaction. In another embodiment, thestatistical information relates to prior interactions of one or moreother users. In an example, an interface presents Button A and Button Bproximate to one another, and statistical information indicates thatninety percent of interactions by all users are with Button A, while tenpercent are with Button B. In this case, given inferred coordinates fora particular user located between Button A and Button B, input program106 determines that the inferred interaction is an interaction withButton A.

In some embodiments, the statistical information identifies one or moretrends in the differences between the inferred coordinates of priorinteractions of the user and the confirmed interactions of the user. Inone embodiment, such a trend indicates a statistical relationshipbetween one or more dimensions of the inferred interactions of a userwith the corresponding dimensions of confirmed interactions of the user.For example, a trend for a user that tends to miss elements of atwo-dimensional interface by performing gestures too low indicates thatthe y value of the inferred coordinates tends to be less than the yvalue of the confirmed interactions. Based on such a trend, inputprogram 106 determines that the inferred interaction is more likely tobe at coordinates with a y value greater than the inferred coordinates.

In some embodiments, the statistical information defines one or moreregions, each of which are, in various examples, within, partiallywithin, or outside of the boundaries of one or more interface elements.For each such region, the statistical information includes a probabilitythat an interaction within the region is intended for a particularinterface element. For example, a first region partially overlaps afirst interface element and second interface element, and the firstregion is associated with a 50% probability that an interaction withinthe first region is intended for the first interface element, a 40%probability for the second interface element, and a 10% probability fora third interface element. In one embodiment, statistical information iscompiled based on prior usage information, including prior confirmationsof inferred interactions (see decision operation 214 and operation 216).

In operation 208, input program 106 determines a confidence score forthe inferred interaction. The confidence score represents a probabilitythat the inferred interaction is the interaction intended by the user.In various embodiments, input program 106 determines the confidencescore based on some or all of the following: the proximity of theinferred coordinates to an interface element corresponding to theintended interaction, statistical information, the number of receivedinteractions, and the variance among the received interactions. Forexample, input program 106 determines a high confidence score based onthe inferred coordinates being highly proximate to the interface elementof the intended interaction, the number of received interactions beinghigh, and the variance among the received interactions being low.

In decision operation 210, input program 106 determines whether theconfidence score is above a threshold. In one embodiment, the thresholdis a pre-determined threshold. For example, the threshold ispre-programmed or, alternatively, user-configured. In anotherembodiment, the threshold is a dynamic threshold generated and/orrefined by, for example, machine learning. If input program 106determines that the confidence score is above the threshold (decisionoperation 210, YES branch), then input program 106 presents the inferredinteraction (operation 212). If input program 106 determines that theconfidence score is not above the threshold (decision operation 210, NObranch), then input program 106 receives additional interactions (seeoperation 202) and proceeds as follows thereafter in order to infer aninteraction with a higher confidence score.

In operation 212, input program 106 presents the inferred interaction.In one embodiment, input program 106 presents the inferred interactionby indicating the interface element of the inferred interaction viainterface 104. In various examples, input program 106 indicates theinterface element by modifying the position, size, color, or pattern ofthe interface element. In another embodiment, input program 106indicates the interface element by modifying interface 104 with visualand/or auditory information. In various examples, input program 106indicates the interface element by text or auditory information toidentify the interface element, or by any other visual or auditorymodification of interface 104. In other embodiments, input program 106presents the inferred interaction by an interface other than interface104. In some embodiments, input program 106 temporarily lengthens theinput window in order to allow the user additional time to view thepresented inferred interaction. In some embodiments, input program 106continues to present the inferred interaction either until input program104 determines that negative confirmation has been received (seedecision operation 214) or until input program 104 determines that theconfidence score of the inferred interaction does not exceed thethreshold (see decision operation 210).

In decision operation 214, input program 106 determines whetherconfirmation has been received. In one embodiment, input program 106determines that confirmation has been received based on a confirmationthat the presented inferred interaction is the interaction intended bythe user. In various embodiments, the confirmation is implicit orexplicit. Input program 106 determines that implicit confirmation hasbeen received based on receiving no additional interactions within aninput window.

Input program 106 determines that explicit confirmation has beenreceived based on receiving an interaction with a confirmation interfaceelement. For example, input program 106 presents the inferredinteraction (see operation 212) by displaying (e.g., via interface 104)a confirmation interface element (e.g., a dialog box, interface button,or audio prompt), which prompts the user to confirm whether the inferredinteraction is the interaction intended by the user. Input program 106receives confirmation based on a user interaction with the confirmationinterface element.

In some embodiments, input program 106 receives negative confirmation toindicate that the presented inferred interaction does not match theinteraction intended by the user. In one such embodiment, a confirmationinterface element allows the user to express either a positiveconfirmation or a negative confirmation. For example, the confirmationinterface element is a dialog box that prompts the user to confirmwhether the inferred interaction was inferred correctly, and provides a“yes” button representing a positive confirmation and a “no” buttonrepresenting a negative confirmation.

In various embodiments, input program 106 receives confirmation by anyinput means, including, but not limited to: interface 104, a peripheraldevice (e.g., mouse, keyboard, assistive device), or auditory (e.g.,spoken) input.

If input program 106 determines that confirmation has not been received(decision operation 214, NO branch), then input program 106 receivesadditional interactions (see operation 202) and proceeds as followsthereafter. In some embodiments, input program 106 ceases presentationof the inferred interaction. In other embodiments, input program 106continues presentation of the inferred interaction. For example, inputprogram 106 continues to present an inferred interaction that wasimplicitly negatively confirmed. Alternatively, input program 106 ceasesto present an inferred interaction that was expressly negativelyconfirmed. In one embodiment, input program 106 disregards an interfaceelement of an inferred interaction that was expressly negativelyconfirmed when determining a subsequent inferred interaction. Forexample, input program 106 disregards the such an interface element,thereby preventing the suggestion of the same inferred interactionmultiple times after a user has already rejected it as the interactionintended by the user.

If input program 106 determines that confirmation has been received(decision operation 214, YES branch), then input program 106 determinesthe confirmed interaction to be the inferred interaction (operation216). In some embodiments, input program 106, in response to determiningthe confirmed interaction, performs the confirmed interaction viainterface 104. For example, input program 106 performs an interactionwith an interface element in response to determining that the confirmedinteraction is the interaction with the interface element.

FIG. 3 is an illustrative example of a user interface, on a computingdevice within the environment of FIG. 1, in accordance with anembodiment of the present disclosure. For example, FIG. 3 is anillustrative example of a user interface, generally designated 300, oncomputing device 102 within computing environment 100 of FIG. 1.

Interface 300 is presented (e.g., displayed) to a user of computingdevice 102. Interface 300 is an example of interface 104. Interface 300includes element group 310, element group 320, and confirmation inputelement 330. Element group 310 includes interface elements 311, 312,313, and 314, each of which is an interactive user interface element.Element group 320 includes interface elements 321, 322, 323, 324, 325,and 326, each of which is an interactive user interface element.

In this example, a user of computing device 102 attempts a firstinteraction and a second interaction with interface 300. The firstinteraction that the user attempts is an interaction with input element312. Thus, the intended interaction is an interaction with input element312. In this example, the user attempts the first interaction with fivesuccessive gestures before confirming a presented inferred interaction,as will be described below.

Input program 106 receives interaction 319 a based on the userperforming a gesture (see operation 202). Input program 106 determinesinferred coordinates based on the coordinates of interaction 319 a (seeoperation 204). Input program 106 determines an inferred interactionwith interface element 311 (see operation 206). Input program 106determines a confidence score for the inferred interaction (seeoperation 208). In response to determining that the confidence scoredoes not exceed a pre-determined threshold (see decision operation 210),input program 106 receives additional interactions (see operation 202).

Input program 106 receives interaction 319 b based on the userperforming another gesture (see operation 202). Input program 106determines inferred coordinates based on the coordinates of interaction319 a and 319 b (see operation 204). Input program 106 determines aninferred interaction with interface element 312 (see operation 206).Input program determines a confidence score for the inferred interaction(see operation 208). Based on the confidence score exceeding apre-determined threshold (see decision operation 210), input program 106presents the inferred interaction with interface element 312 (seeoperation 212). Input program 106 determines whether confirmation hasbeen received (see decision operation 216). Input program 106 candetermine that confirmation has been received based on, for example,input program 106 receiving no additional inputs within an input window,or input program 106 receiving an interaction with confirmation inputelement 330.

However, in this example, the user continues to perform gestures, soinput program 106 determines that no confirmation of the presentedinferred interaction was received (see decision operation 214). Inputprogram 106 receives interactions 319 c, 319 d, and 319 e based on theuser performing additional gestures (see operation 202). For each, inputprogram 106 proceeds as described above in connection with each ofinteractions 319 a and 319 b. In various embodiments, based on eachadditional interaction, input program 106 updates some or all of thedetermination of the inferred coordinates, the determination of theinferred interaction, and the confidence score. For example, updatingthe inferred interaction includes re-computing a geometric average basedon the one or more received interactions, including any additionalinteractions. Input program 106 continues to present an inferredinteraction with interface element 312 (see operation 212). Based oninput program 106 receiving no further interactions (after interaction319 e) within an input window, input program 106 determines that thepresented inferred interaction with interface element 312 is implicitlyconfirmed (see decision operation 214). Input program 106 performs theconfirmed interaction with interface element 312 (see operation 216).

The second interaction is an interaction with interface elements 323,324, and 325. The user attempts the second interaction with threesuccessive, continuous gestures. Input program 106 receives interaction329 a (see operation 202). Input program 106 determines inferredcoordinates based on the coordinates of interaction 329 a (see operation204). Input program 106 determines an inferred interaction withinterface elements 324, 325, and 326 (see operation 206). Input program106 determines a confidence score for the inferred interaction (seeoperation 208). In response to determining that the confidence scoreexceeds a pre-determined threshold (see decision operation 210), inputprogram 106 presents the inferred interaction (see operation 212). Theuser views the inferred interaction and performs another continuousgesture, thereby implicitly rejecting the inferred interaction. Inputprogram 106 determines that no confirmation has been received for theinferred interaction based on receiving interaction 329 b (see decisionoperation 214).

Input program 106 determines inferred coordinates based on interactions329 a and 329 b (see operation 204). Input program 106 determines aninferred interaction with interface element 324 and 325 based in part,in this example, on a determination that the inferred interaction wouldnot select either interface element 323 or 326 (see operation 206).Input program 106 determines a confidence score for the inferredinteraction (see operation 208). In this example, input program 106determines a low confidence score based on the length of the receivedinteraction beyond the boundaries of the interface elements selected bythe inferred interaction (i.e., interface elements 324 and 325). Inputprogram 106 determines that the confidence score does not exceed thepredetermined threshold (see decision operation 210). In response, inputprogram 106 ceases to present the inferred interaction (see operation212). The user observes that no inferred interaction is presented andmakes a third continuous gesture. In this example, all three gesturesoccurred within the input window.

Input program 106 determines inferred coordinates based on interactions329 a, 329 b, and 329 c (see operation 204). Input program 106determines an inferred interaction with interface elements 324, 325, and323 (see operation 206). Input program 106 determines a confidence scorefor the inferred interaction (see operation 208). Input program 106determines that the confidence score exceeds the predetermined threshold(see decision operation 210). In response, input program 106 presentsthe inferred interaction (see operation 212). The user observes theinferred interaction and performs a gesture to interact withconfirmation input 130. Input program 106 determines that confirmationof the inferred interaction has been received based on input program 106receiving an interaction with confirmation input 130 (see decisionoperation 214). Input program 106 performs the confirmed interaction,being an interaction with interface elements 324, 325, and 323 (seeoperation 216).

FIG. 4 depicts a block diagram of components of computing device 102, inaccordance with an illustrative embodiment of the present invention. Itshould be appreciated that FIG. 4 provides only an illustration of oneimplementation and does not imply any limitations with regard to theenvironments in which different embodiments may be implemented. Manymodifications to the depicted environment may be made.

Computing device 102 includes communications fabric 402, which providescommunications between computer processor(s) 404, memory 406, persistentstorage 408, communications unit 410, and input/output (I/O)interface(s) 412. Communications fabric 402 can be implemented with anyarchitecture designed for passing data and/or control informationbetween processors (such as microprocessors, communications and networkprocessors, etc.), system memory, peripheral devices, and any otherhardware components within a system. For example, communications fabric402 can be implemented with one or more buses.

Memory 406 and persistent storage 408 are computer-readable storagemedia. In this embodiment, memory 406 includes random access memory(RAM) 414 and cache memory 416. In general, memory 406 can include anysuitable volatile or non-volatile computer-readable storage media.

Interface 104 and input program 106 are stored in persistent storage 408for execution and/or access by one or more of the respective computerprocessors 404 via one or more memories of memory 406. In thisembodiment, persistent storage 408 includes a magnetic hard disk drive.Alternatively, or in addition to a magnetic hard disk drive, persistentstorage 408 can include a solid state hard drive, a semiconductorstorage device, read-only memory (ROM), erasable programmable read-onlymemory (EPROM), flash memory, or any other computer-readable storagemedia that is capable of storing program instructions or digitalinformation.

The media used by persistent storage 408 may also be removable. Forexample, a removable hard drive may be used for persistent storage 408.Other examples include optical and magnetic disks, thumb drives, andsmart cards that are inserted into a drive for transfer onto anothercomputer-readable storage medium that is also part of persistent storage408.

Communications unit 410, in these examples, provides for communicationswith other data processing systems or devices, including resources ofnetwork 4XX. In these examples, communications unit 410 includes one ormore network interface cards. Communications unit 410 may providecommunications through the use of either or both physical and wirelesscommunications links. Interface 104 and input program 106 may bedownloaded to persistent storage 408 through communications unit 410.

I/O interface(s) 412 allows for input and output of data with otherdevices that may be connected to computing device 102. For example, I/Ointerface 412 may provide a connection to external devices 418 such as akeyboard, keypad, a touch screen, and/or some other suitable inputdevice. External devices 418 can also include portable computer-readablestorage media such as, for example, thumb drives, portable optical ormagnetic disks, and memory cards. Software and data used to practiceembodiments of the present invention, e.g., interface 104 and inputprogram 106, can be stored on such portable computer-readable storagemedia and can be loaded onto persistent storage 408 via I/O interface(s)412. I/O interface(s) 412 also connect to a display 420.

Display 420 provides a mechanism to display data to a user and may be,for example, a computer monitor, or a television screen.

The programs described herein are identified based upon the applicationfor which they are implemented in a specific embodiment of theinvention. However, it should be appreciated that any particular programnomenclature herein is used merely for convenience, and thus theinvention should not be limited to use solely in any specificapplication identified and/or implied by such nomenclature.

The term(s) “Smalltalk” and the like may be subject to trademark rightsin various jurisdictions throughout the world and are used here only inreference to the products or services properly denominated by the marksto the extent that such trademark rights may exist.

The present invention may be a system, a method, and/or a computerprogram product. The computer program product may include a computerreadable storage medium (or media) having computer readable programinstructions thereon for causing a processor to carry out aspects of thepresent invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Smalltalk, C++ or the like, andconventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server. In the latter scenario,the remote computer may be connected to the user's computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection may be made to an external computer(for example, through the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) may execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitry,in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

The descriptions of the various embodiments of the present inventionhave been presented for purposes of illustration, but are not intendedto be exhaustive or limited to the embodiments disclosed. Manymodifications and variations will be apparent to those of ordinary skillin the art without departing from the scope and spirit of the invention.The terminology used herein was chosen to best explain the principles ofthe embodiment, the practical application or technical improvement overtechnologies found in the marketplace, or to enable others of ordinaryskill in the art to understand the embodiments disclosed herein.

What is claimed is:
 1. A computer program product comprising a computerreadable storage medium having program instructions embodied therewith,wherein the computer readable storage medium is not a transitory signalper se, the program instructions executable by a processor to cause theprocessor to perform a method comprising: receiving one or moreinteractions from a user via a user interface, wherein the one or moreinteractions include a touch gesture performed on a touch screen;determining an inferred interaction based, at least in part, on ageometric average of coordinate dimensions of the one or moreinteractions, one or more weights based on a closeness of the coordinatedimensions to an edge of the user interface, and statistical informationcorresponding to the user interface; determining a probability that theinferred interaction is an interaction intended by the user; in responseto determining that the probability is above a pre-determined threshold,presenting the inferred interaction via the user interface; determiningwhether a confirmation has been received for the inferred interaction byidentifying the inferred interaction via the user interface, presentingan interface element via the user interface, the interface elementprompting the user for the confirmation of the inferred interaction, andreceiving an interaction via the interface element; in response todetermining that the confirmation has been received for the inferredinteraction, determining that the confirmation is a positiveconfirmation; in response to determining that the confirmation is apositive confirmation, performing the inferred interaction; receivingone or more additional interactions from the user via the userinterface; and updating the inferred interaction based on a geometricaverage of the coordinate dimensions of the one or more interactions andcoordinate dimensions of the one or more additional interactions, one ormore weights based on a closeness of the coordinate dimensions of theone or more additional interactions to an edge of the user interface,and additional statistical information corresponding to the userinterface.